While virtual reality (VR) opens up exciting opportunities in several areas including content creation, and sports, entertainment and game broadcasting, VR applications also bring new challenges when attempting to deliver immersive experiences to a broad base of users
Geometrical Acoustic (GA) processing is the simulation of sound ray propagation in a particular spatial model (e.g., a virtual scene setting in a VR game), which can be executed by a GA processing pipeline for example. Based on geometrical information about the setting, GA processing can automatically determine how the sound waves travel and bounce around the environment and reach a character or another type of object in the setting (e.g., which is subject to control by a player in real-time), thus producing 3D spatialized audio data.
Typically, a GA processing pipeline processes the geometry of a virtual scene along with knowledge of sound sources and receiver location by using a ray tracing algorithm and an audio rendering algorithm. The ray tracing algorithm is used to compute a spatial acoustic model and generate impulse responses (IRs) that encode the delays and attenuation of sound waves traveling from a sound source object to a sound receiver object through different propagation paths representing transmission, reflection, and diffraction. Rays (or sound waves) are traced to generate an impulse response which represents the decay of audio energy in time at the place of the sound receiver. Whenever the sound source, the receiver, or another object in the scene moves, these propagation paths need to be recomputed—sometimes periodically. The audio rendering algorithm is used to generate audio signals by convolving the input audio signals with the IRs. In a virtual environment such as in a game, where the acoustic geometry of the scene is known along with the positions of a sound source and a sound receiver, GA processing is applied to generate spatialized audio data.
There are multiple on-going efforts to have GA audio in computation intensive games to provide an immersive VR experience. But these efforts are restricted to application of GA to only in-game audio, e.g., audio generated by the application program. In existing systems, verbal interaction among multi-players as captured by microphones can be rendered concurrently with the game play. However, the microphone-captured audio can only be rendered in a plain and unprocessed manner and not immersed in, but rather detached from, the virtual scenes. Multiplayer VR applications can be performed by making the interactions of the players and the application more immersive.